Many of today's digital image acquisition devices capture a color image using a color filter array (CFA) and an array of optical sensors that is overlaid by the CFA. The CFA has a plurality of filter elements, each of which overlays one of the optical sensors in the optical sensor array, and each filter element is designed to pass only one color of light. The most common type of CFA is the Bayer pattern filter (also referred to as a Bayer filter). The Bayer filter comprises filter elements that pass three different colors of light: green (G), blue (B), and red (R). These filter elements are arranged in alternating rows of green-blue and red-green. With this pattern, red and blue filter elements appear on every other row and every other column, while green filter elements appear on every row and every column. This pattern exploits the human eye's higher sensitivity to green light.
As noted, each filter element passes light of only one color. Thus, in the Bayer filter, a green filter element passes only green light, a blue filter element passes only blue light, and a red filter element passes only red light. This means that an optical sensor behind a green filter element senses only the green portion of the light that hits the green filter element, an optical sensor behind a blue filter element senses only the blue portion of the light that hits the blue filter element, and an optical sensor behind a red filter element senses only the red portion of the light that hits the filter element. Thus, because of the CFA, the image captured by the sensor array is a mosaic image. As used herein, the term mosaic image refers to an image in which there is only partial color information for each pixel of the image. In the case of a mosaic image captured using a Bayer filter, there is only information for one of the three color channels of green, blue, and red for each pixel.
After capturing a mosaic image through the Bayer filter, the array of optical sensors outputs a plurality of values. Each value comes from one of the optical sensors, and each value corresponds to a particular pixel of the image (each optical sensor represents a pixel). Since each pixel corresponds to one of the optical sensors, and since each optical sensor corresponds to one of the filter elements of the Bayer filter, each pixel in turn corresponds to one of the filter elements of the Bayer filter. For each pixel, only one of the three color values is provided by the optical sensor array. This color value will be referred to herein as the sensed pixel value. For example, the sensed pixel value provided by an optical sensor overlaid with a green filter element for a pixel is for the green color channel. Thus, this pixel is referred to as a green pixel. Similarly, the sensed pixel value provided by an optical sensor overlaid with a blue filter element for a pixel is for the blue color channel; thus, this pixel is referred to as a blue pixel. Similarly, the sensed pixel value provided by an optical sensor overlaid with a red filter element is for a red pixel.
In order to render a pixel in full color, a color value is needed for all three of the color channels. Since only one pixel value (corresponding to one of the three color channels) is provided by the sensor array for each pixel, the other two color values need to be derived for each pixel, which means that for each pixel, there is one pixel value and two derived pixel values (corresponding to the other two color channels). A demosaicing process is performed on the mosaic image to calculate the two derived pixel values for each pixel. Through the mosaicing and demosaicing process, a color image can be captured and reproduced.
The sensed pixel values provided by the sensor array may and most likely will include some noise components. These noise components may be due to various factors, such as environmental conditions, manufacturing variations in the optical sensors, etc., and the amount of noise may vary from sensor to sensor. This noise in the sensed pixel values can lead to the introduction of unwanted, spurious colors in the demosaiced image. For example, if the noise in the sensed pixel value for a blue pixel causes the blue component of the pixel to be too high, it may cause the final demosaiced version of that pixel to take on a color that does not reflect the true color of that pixel in the original image. This can lead to a strange and unpleasant image. To prevent this, most image processing systems perform some form of denoising on the mosaic image.
Denoising is often done by averaging the pixel values of nearby pixels on a color channel by color channel basis. For example, a blue pixel of interest may be denoised by taking the pixel values of neighboring blue pixels and averaging them with the pixel value of the blue pixel of interest to derive a denoised pixel value for the blue pixel of interest. A green pixel may be denoised in the same way by taking into account the sensed pixel values of neighboring green pixels, and a red pixel may be denoised by taking into account the sensed pixel values of neighboring red pixels. Thus, pixels are typically denoised on a color channel by color channel basis (in other words, pixels of different colors are denoised separately based on neighboring pixels of the same color).
Some approaches to denoising include isotropic averaging and Gaussian filter averaging. However, while these “global filtering” approaches perform adequately in denoising relatively “uniform” or “flat” areas of an image, these approaches are generally not acceptable denoising strategies for most images since most images contain “edge areas”, “line areas”, or “detail areas” that would be considerably damaged if these approaches were applied. A “uniform” or “flat” image environment is one in which characteristics of the pixels around a pixel of interest indicate little or no change in the image from one feature to another (e.g., an area of the image depicting a clear sky). An “edge” or “line” image environment is one in which the pixels around the pixel of interest have characteristics that indicate a consistent unidirectional change from one image feature to another (e.g., an area of the image in which there is a change from a parcel box to a table on which the parcel box is placed). A “detail” image environment is one in which the pixels around the pixel of interest have characteristics that indicate many inconsistent multidirectional changes between image features (e.g., an area of the image depicting a floor carpet).
Other approaches such as bilateral filtering and non-local means filtering aim to denoise an image but not at the expense of loosing edges, lines, and details in the image. Generally, these approaches analyze a neighborhood of pixels surrounding a pixel of interest to identify “similar” pixels within the neighborhood. Similar pixels are combined with the pixel of interest in such a way to remove noise from the pixel of interest. The more similar a neighboring pixel is to the pixel of interest, the more the similar pixel is used in denoising the pixel of interest and, hence the more noise that may be reduced from the pixel of interest. However, these approaches are typically computationally intensive and their implementation in hardware would result in a prohibitively high gate count for use in many consumer electronic devices.
What is needed, therefore, is a new, less computationally intensive, approach for effectively denoising an image that preserves edges, lines, and details in the image.